{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Throughtput Benchmarking  Seldon-Core on GCP Kubernetes\n",
    "\n",
    "The notebook will provide a benchmarking of seldon-core for maximum throughput test. We will run a stub model and test using REST and gRPC predictions. This will provide a maximum theoretical throughtput for model deployment in the given infrastructure scenario:\n",
    "  \n",
    "   * 1 replica of the model running on n1-standard-16 GCP node\n",
    "   \n",
    "For a real model the throughput would be less. Future benchmarks will test realistic models scenarios.\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Create Cluster"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": true
   },
   "source": [
    "Create a cluster of 4 nodes of machine type n1-standard-16 \n",
    "\n",
    "```bash\n",
    "PROJECT=seldon-core-benchmarking\n",
    "ZONE=europe-west1-b\n",
    "gcloud beta container --project \"${PROJECT}\" clusters create \"loadtest\" \\\n",
    "    --zone \"${ZONE}\" \\\n",
    "    --username \"admin\" \\\n",
    "    --cluster-version \"1.9.3-gke.0\" \\\n",
    "    --machine-type \"n1-standard-16\" \\\n",
    "    --image-type \"COS\" \\\n",
    "    --disk-size \"100\" \\\n",
    "    --num-nodes \"4\" \\\n",
    "    --network \"default\" \\\n",
    "    --enable-cloud-logging \\\n",
    "    --enable-cloud-monitoring \\\n",
    "    --subnetwork \"default\"\n",
    "```"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Install helm"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "!kubectl -n kube-system create sa tiller\n",
    "!kubectl create clusterrolebinding tiller --clusterrole cluster-admin --serviceaccount=kube-system:tiller\n",
    "!helm init --service-account tiller"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Start Seldon-Core CRD"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "!helm install ../helm-charts/seldon-core-crd --name seldon-core-crd --set usage_metrics.enabled=true"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Cordon off loadtest nodes"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "!kubectl get nodes"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We cordon off first 3 nodes so seldon-core and the model will not be deployed on the 1 remaining node."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "!kubectl cordon $(kubectl get nodes -o jsonpath='{.items[0].metadata.name}')\n",
    "!kubectl cordon $(kubectl get nodes -o jsonpath='{.items[1].metadata.name}')\n",
    "!kubectl cordon $(kubectl get nodes -o jsonpath='{.items[2].metadata.name}')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Label the nodes so they can be used by locust."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "!kubectl label nodes $(kubectl get nodes -o jsonpath='{.items[0].metadata.name}') role=locust\n",
    "!kubectl label nodes $(kubectl get nodes -o jsonpath='{.items[1].metadata.name}') role=locust\n",
    "!kubectl label nodes $(kubectl get nodes -o jsonpath='{.items[2].metadata.name}') role=locust"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Start seldon-core"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "!helm install ../helm-charts/seldon-core --name seldon-core \\\n",
    "        --set apife.enabled=true \\\n",
    "        --set engine.image.tag=0.1.6_SNAPSHOT_loadtest \\\n",
    "        --set cluster_manager.image.tag=0.1.6_SNAPSHOT_loadtest\n",
    "        "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Wait for seldon-core to start"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "!kubectl get pods -o wide"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Create Stub Deployment"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "!pygmentize resources/loadtest_simple_model.json"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "!kubectl apply -f resources/loadtest_simple_model.json"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Wait for deployment to be running."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "!kubectl get seldondeployments seldon-core-loadtest -o jsonpath='{.status}'"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Run benchmark"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Uncorden the first 3 nodes so they can be used to schedule locust"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "!kubectl uncordon $(kubectl get nodes -o jsonpath='{.items[0].metadata.name}')\n",
    "!kubectl uncordon $(kubectl get nodes -o jsonpath='{.items[1].metadata.name}')\n",
    "!kubectl uncordon $(kubectl get nodes -o jsonpath='{.items[2].metadata.name}')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## gRPC\n",
    "Start locust load test for gRPC"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "!helm install ../helm-charts/seldon-core-loadtesting --name loadtest  \\\n",
    "    --set locust.host=loadtest:5001 \\\n",
    "    --set locust.script=predict_grpc_locust.py \\\n",
    "    --set oauth.enabled=false \\\n",
    "    --set oauth.key=oauth-key \\\n",
    "    --set oauth.secret=oauth-secret \\\n",
    "    --set locust.hatchRate=1 \\\n",
    "    --set locust.clients=256 \\\n",
    "    --set loadtest.sendFeedback=0 \\\n",
    "    --set locust.minWait=0 \\\n",
    "    --set locust.maxWait=0 \\\n",
    "    --set replicaCount=64 "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "To download stats use \n",
    "\n",
    "```bash\n",
    "if [ \"$#\" -ne 2 ]; then\n",
    "    echo \"Illegal number of parameters: <experiment> <rest|grpc>\"\n",
    "fi\n",
    "\n",
    "EXPERIMENT=$1\n",
    "TYPE=$2\n",
    "\n",
    "MASTER=`kubectl get pod -l name=locust-master-1 -o jsonpath='{.items[0].metadata.name}'`\n",
    "\n",
    "kubectl cp ${MASTER}:stats_distribution.csv ${EXPERIMENT}_${TYPE}_stats_distribution.csv\n",
    "kubectl cp ${MASTER}:stats_requests.csv ${EXPERIMENT}_${TYPE}_stats_requests.csv\n",
    "```"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "You can get live stats by viewing the logs of the locust master"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "!kubectl logs $(kubectl get pod -l name=locust-master-1 -o jsonpath='{.items[0].metadata.name}') --tail=10"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "!helm delete loadtest --purge"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## REST \n",
    "Run REST benchmark"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "!helm install ../helm-charts/seldon-core-loadtesting --name loadtest  \\\n",
    "    --set locust.host=http://loadtest:8000 \\\n",
    "    --set oauth.enabled=false \\\n",
    "    --set oauth.key=oauth-key \\\n",
    "    --set oauth.secret=oauth-secret \\\n",
    "    --set locust.hatchRate=1 \\\n",
    "    --set locust.clients=256 \\\n",
    "    --set loadtest.sendFeedback=0 \\\n",
    "    --set locust.minWait=0 \\\n",
    "    --set locust.maxWait=0 \\\n",
    "    --set replicaCount=64"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Get stats as per gRPC and/or monitor"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "!kubectl logs $(kubectl get pod -l name=locust-master-1 -o jsonpath='{.items[0].metadata.name}') --tail=10"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "!helm delete loadtest --purge"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "!kubectl cordon $(kubectl get nodes -o jsonpath='{.items[0].metadata.name}')\n",
    "!kubectl cordon $(kubectl get nodes -o jsonpath='{.items[1].metadata.name}')\n",
    "!kubectl cordon $(kubectl get nodes -o jsonpath='{.items[2].metadata.name}')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Tear Down"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "!kubectl delete -f resources/loadtest_simple_model.json"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "!helm delete seldon-core --purge"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "!helm delete seldon-core-crd --purge"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.4"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 1
}
